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Abstract: We study the robustness of optima] regulators for nonlinear, deterministic and stochastic, 
multi-input, dynamical systems, under the assumption that all state variables can be measured. 
We show that, under mild assumptions, such nonlinear regulators have a guaranteed infinite gain 
margin: moreover, they have a guaranteed 50 percent gain reduction margin and a 60 degree phase 
margin, in each feedback channel, provided that the system is linear in the control and the penalty 
to the control is quadratic, thus extending the well-known properties of LQ regulators to nonlinear 
optimal designs. These results are also valid for infinite horizon, average cost, stochastic optimal 
control problems. 
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Regulator design for dynamical systems is usually performed on the basis of a nominal model 
of the plant to be controlled. Modelling errors are unavoidable and, in fact, often desirable because 
they may result in simpler designs. i* is therefore essential that the regulator based on the nominal 
model is robust; that is, it preserves its qualitative properties (namely, the stability of the closed- 
loop system) in the face of modelling errors. 

The robustness and sensitivity to modelling errors of controlled linear systems has been exten- 
sively studied in the past [2. 6). The robustness (stability margins) of regulators has been traditionally 
described in terms of gain and phase margins, although more recent approaches [3, 9, 12] focus on 
the singular values of the return difference or of the inverse return difference matrix. 

One of the most appealing features of optimal linear quadratic (LQ) regulators are their 
guaranteed stability margins. Namely, LQ regulators remain stable when the control gains are mul- 
tiplied by any number greater than 1/2. They also have guaranteed phase margins of sixty degrees 
[1, 13, 14, 16], These results can be obtained directly by appropriately manipulating the associated 
Riccati equation [13]. 

A recent paper by Glad [5] has shown that gain margins of optimal regulators for nonlinear 
systems can be derived from the associated Hamilton- Jacobi-Bellman (HJB) equation, under suitable 
assumptions. This result ties nicely with the results on LQ regulators, because the Riccati equation 
is a direct consequence of the HJB equation associated with LQ problems. However, the results of [5] 
are only applicable to single-input, deterministic systems, perturbed by memoryless non linearities 
thus allowing only derivation of gain margin results; no phase margin results were derived in [5]. 

In this paper we derive general robustness margins of optimal regulators for multi-input nonlinear 
systems. Our results are valid for both deterministic and stochastic systems (controlled diffusion 
processes). In contrast to [5], we allow dynamical, (i.e. not j»*3t memory less) perturbations inside the 
loop and obtain, as a corollary, a generalization of the phase margin results of [13]. In particular, 
we show (Theorem 3) that the robustness margins of LQ regulators (including the 60 degree phase 
margin) hold for optimal regulators of any nonlinear plant which is linear in the control, provided 
that the cost functional contains a quadratic control penalty. 

In the stochastic case, we consider two distinct classes of controlled processes: a) Those for which 
the state can be steered to an equilibrium point (assumed to be the origin). Such is the case for 
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diffusion procesf& ft wbsctoi thr intensity of the noise decreases to zero as the equilibrium point is 
approached. We then consider the associated infinite horizon, expected total cost, optimal control 
problem, b) Those for which the intensity of the noise is allowed to be everywhere positive. (The 
LQG problem with perfect observations is an example.) In that case no control law can achieve 
finite total cost; we consider, however, the associated infinite horizon, expected average cost, optimal 
control problem. We then derive the same results, provided that stability is now given an appropriate 
meaning: that no sample path converges to infinity. 

We reiterate that the above robustness results are only valid for nonlinear optimal control 
problems in which all state variables can be measured exactly and can be used in the implementation 
of the nonlinear feedback regulator. Robustness properties of nonlinear stochastic regulators that 
arise when only noisy measurements of output variables are available are not addressed in this paper; 
they remain the subject of future research. 


II. PROBLEM FORMULATION. 

Notation: Throughout this paper, scalar functions will be indicated by lowercase letters; vector 
functions by lowercase underlined letters; matrix functions by uppercase letters. For any vector 
function / wc will use subscripts (e g. /,) to denote its scalar components. For an}’ scalar function / 
of a vector input x, we let df/dx denote the transpose of the gradient of / (a row vector). 

Case A: Deterministic Optimal Control. 

Consider the controlled deterministic system 

§(<) = /(!(<). mM); j(0) = Ioi (l) 

where x, a are n- and m-dimensional state and control vectors, respectively, and / is a continuous 
function from R n+m into R n such that /( 0, 0) = 0. A control law is a measurable function & R m 
such that the closed-loop equation 

f t (o = mt), mm, i («) = m 

has a unique solution, for all JoGR”. (If &(■) is not continuous some more care may be needed in 
defining what is meant by a solution to (2); see [4].) Let l:R n ►-» R and h:R m R be nonnegative 
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measurable functions denoting the penalties to the state and the control, respectively, satisfying 
1(0) = fip) = 0. We consider the performance criterion 

A = [ mt)) + h(u(t)))dt. (3) 

Jo 

The genera] dynamic programming conditions for optimality for such control problems are well- 
known and easy to establish formally. However, for our purposes, we do not need to concern ourselves 
with the particular assumptions that can guarantee existence of optimal control laws or that the 
Hamilton- Jacobi-Bellrnan (HJB) equation is satisfied. Such issues are treated, for example, in [4] for 
finite horizon problems. We will assume instead that the data of the control problem are sufficiently 
well-behaved to guarantee that no complication will arise. In particular, we assume: 

Assumption la: There exists an optimal control law £(•). Moreover, the optimal cost-to-go (value) 
function V:R : ‘ •-» R is continuously differentiable and satisfies the HJB equation 

° = ^(") + + + *(*)» Vu€i? rn . (4) 

Finally, V'(x) > 0, Vx^O and lim infj; ^ V(i) > 0. 

Case B: Stochastic Optimal Control; Total Cost. 

Consider the perfectly observed controlled diffusion process 

di(t) = f{s.{t),u(t))dt + L(x[t))dui(i); s(0) — i,,, (5) 

where z, u, / are as in Case A, except that / is now allowed to be any measurable function; £(x) 
is a measurable n X n matrix function, m(t) is a standard n-dimensional Brownian motion and Zo 
is the initial state We also assume that «;(f) is defined on some probability space (ft,?, 9) and we 
denote by the smallest o-field in 9 such that w(r) is 9 t -measurable, for all r<t. 

A control law is a measurable function &.7i n R m such that the stochastic differential equation 

dz{t) = f_{z{t), b(z(t))) dt + £(i(0) dw.{t); s(0) = ^ (6) 
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has a unique solution in the Ito sense [15], We consider the performance criterion 


Um E\[ (l{x(t))+h(u(tm 

r-oc J o 


( 7 ) 


where /(-) and h ( ■) are as for Case A. Let E r (x) denote the transpose of E(x). Let A(x) = ( l/2)E(2)E r (x) 
and a,j(i ) bt the t, j-th entry of A(x). We define a differential operator L * by 




i=i 


i.j=i 


dx{dxj 


( 8 ) 


As in the deterministic case, we will assume: 

Assumption lb: There exists an optimal control law k ( •). Moreover, the optimal cost-to -go (value) 
function V:R n >-* R is twice continuously differentiable and satisfies the HJB equation 

0 = (L^V)(x) + h{k{x)) + l(x) < [LW){x) + h(u) + /(*), Vx&T\ VuG/T. (9) 

Finally, V'(x) > 0, Vx=^0 and lim infij,||_ 3C V{x) > 0. 


Conditions under which assumption lb is satisfied may be obtained along the lines of [4,7,10] 
and thev need not concern us here. 


Case C: Stochastic Optimal Control; Average Cost. 

Let everything be as in Case B above, except that the performance criterion is modified to be 


Jj- Jim =EI 

T — oo 1 


s: 




( 10 ) 


We also require that {x:l(z) < c} is bounded, for any constant c. 
Assumption lb must then be modified as foiiows:[8] 


Assumption 1c: There exists an optimal control law &(■), a constant g and a function V:R n t-» R 
which is twice continuously differentiable and satisfies the HJB equation 

9 = (L^VX!) + h(t if)) + 1(f) < (Z.“V)(!) + h(u) + 1(f), V u eR m (11) 
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Suppose that the optimal regulator u[t) — k{z(t)), for any of the problems A, B, or C above, is 
perturbed to u(t) -- $(fc(x( )))(i), as in Figure 1. We are interested in the stability of the new closed- 
loop system under suitable assumptions on <t>. The perturbation 4> may be simply a memoryless 
nonlinearity, in which case we can make statements about the gain margins of the optimal regulator. 
It can also be a causal dynamical operator (e.g. a linear time-invariant system); in particular if $ 
corresponds to a pun phase shift, we can make statements about the phase margins of the optimal 
regulator. 

We now proceed to define the class of admissible perturbations 4>. Let &(-) be a control law for 
any of the problems A, B or C. Let -It denote the set of measurable m-dimensional time functions 
from [0, oo) into R. An admissible perturbation 4> of k( ) is a map from Jk into Jl such that: 

Case A: (Deterministic Systems) There exists some u()EJt> such that 

(i) The differential equation (1) has a unique solution x(-). 

(ii) u{t) = $( 4 U( )))(0, VI. 

Cases B and C: (Stochastic Systems) There exists a measurable stochastic process u(t) defined on 
(a*?, 9) such ^hat: 

(i) u(t) is adapted to {5 ( }. 

(ii) The stochastic differential equation (5) has a unique solution x(-). 

(iii) For any sample path, u(t) — <P(k(z[-)))(t). 

Assumption 2: Let jtlo -= {u( )C>^ - f^°h(u(T))dT < oo}. Then, <l> maps into Jbo- 
(For example, if h is a quadratic function, $ must map L>2 into L2 ) 


The solution z{t) of either equation (1) or (5), when u(t) is given as in the above definition, will 
be called the ' trajectory of the perturbed closed-loop system”. 


Ill , .MAJIi-RESIILXS... 

Our first result is a multi-loop generalization of Theorem 3 of [5] which also covers Btochastic 
control problems. It shows that optimal regulators have an infinite gain margin, provided that the 
following condition is satisfied: 

Assumption 3: (i) £(z, u} is differentiable with respect to u, for any fixed zQR n . 
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(ii) For any fixed zGR’\ a€R", m}, either 


£-li T Hlu))>0. Vu SR", or (12) 

■ ttl,u))<», VM6H™. (13) 

Assumption 4: For each t, there exist functions <£,(-, t):R ►-+ R such that 

+(u( ))(l) = Wl(ui(t),0. 0) r , Vii( )6^,. (14) 

Moreover, these functions satisfy, for each t, the sector condition (Figure 2) 

e 2 <*iM). VcGR, Vi. (15) 

In other wor N the perturbation <J> corresponds to a memoryless nonlinearity and, in particular, to 
a gain increase. 

Theorem 1: Consider the optimal control problems A, B, C and let assumptions la, lb, lc, respec- 
tively, as well as 3, hold and suppose that h(y) = YlT—i M u «')i f° r appropriate scalar functions h,-. 
Let $ be an admissible perturbation of a corresponding optimal control law, satisfying assumptions 
2 and 4, and let z[t) denote the trajectory of the perturbed closed-loop system. Then, 

Case A: (Deterministic Problems) Hindoo g(t) = 0. 

Case B: (Stochastic Total Cost Problems) limt_ooZ(t) = 0, almost surely. 

Case C (Stochastic Average Cost Problems) No sample path converges to infinity, almost surely. 
Thus, in all cases the perturbed nonlinear closed-loop system remains stable. 

Proof: All proofs can be found in the appendix. 

We now discuss the crucial assumption 3. Theorem 1 remains true even if f(x, ■) is not 
differentiable, provided that assumption 3(ii) is appropriately modified, as in [5], (although the 
more general version is more obscure). However, the proof of Theorem 1 reveals that it cannot be 
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significantly further weakened. Assumption 3 essentially guarantees that the (expected) direction 
of motion is still a descent dir:ction (with respect to the value function V) under an arbitrary gain 
increase. Given the importance of Theorem 1, it is a natural question to find particular cases for 
which assumption 3 holds. Glad [5] provides the example (for the single input case) 

/(l, ti) = /‘(s) + &(t)/ 2 (l, u), (16) 

where f x \R n ► R n , R n , f 2 :R n ’*~ l >-* R and where /^(r, ■) is monotonic in u, for any fixed 

£. Interestingly enough, the above example covers all cases allowed by assumption 3 and a similar 
characterization can be also obtained for the multi-input case. This is the subject of the next 
theorem, in which we assume that is twice continuously differentiable with respect to u, because 
this allows a significant simplification of the proof. 

Theorem 2: Let / be twice continuously differentiable with respect to jx, for any fixed £. Then, / 
satisfies assumption 3(ii) if and only if, for any fixed i, it is of the form 

i(n) = <,*+'£ Mu) (IT) 

t 

where f k R m >-* R. Moreover, for any component tx, of jx, at most one of the scalar functions 
may depend on tx,. Finally, each function / * is either increasing in tx,, for all u, or decreasing in tx,, 
for all u- (That is the scalar functions /* satisfy themselves assumption 3. However, the way that 
components are split to form the sum in (17) may change with £.) 

As in [5] more assumptions on the dynamics are needed to obtain more specific robustness 
margins. In what follows we assume that the dynamics are linear in the control: 

Assumption 5: /(£,!*) = £°(i) +^( 1 ) 14 , where £\R n »-> R n and F(s) is a n X m matrix function, for 
each i£R n . 

The next assumption describes the set of perturbations $ that will be allowed. It may seem 
counterintuitive as stated below in its full generality. In fact it is a generalization of the conditions 
imposed in either [5] or [13] as will be shown later. 


7 



*. J* * 


ORIGINAL PMC f§ 
Of POOR QUALITY 


Assumption 6: (i) h( ) is continuously differentiable. 

(ii) There exists some e > 0 such that for any measurable m-dimensional time function u(), and 
for any t>0, 


Jo |V 


o 1 M + tt - tmr)) 


dr> 0. 


(18) 


Theorem 3: Consider the optimal control problems A, B, C and let assumptions la, lb, 1c, respec- 
tively, as well as 5, hold. Let 4> be an admissible perturbation of a corresponding optimal control 
law, satisfying assumptions 2 and 6, and let i{t) denote the trajectory of the perturbed closed-loop 
system. Then, 

Case A: (Deterministic Problems) lim^oo i{t) = 0. 

Case B: (Stochastic Total Cost Problems) lim/_ 3C z(t) = 0, almost surely. 

Case C: (Stochastic Average Cost Problems) No .ample path conveiges to infinity, almost surely. 
Thus, in all cases the perturbed nonlinear closed-loop system remains stable. 

In order to apply Theorem 3, one mainly needs to verify that assumption 6 holds. This is 
done below for certain particular problems. Proposition 1 shows that the robustness margins of LQ 
regulators generalize to nonlinear systems which are linear in the control and in which the penalty 
to the control is quadratic. 

Proposition 1: Suppose that /i(ji) = r, ttjf (r, > 0) and let d> be a linear time invariant system 

with diagonal transfer matrix whose nonzero entries <f>i are proper, stable rational functions and, for 
some e > 0, Re[^,-(jti/)]>l/2 + e, Vo/. Then, assumption 6 holds. The condition Re[^j(jc«/)]>l/2 + e 
is satisfied, in particular, if for some e > 0 

(i) 4> is a memoryless gain, larger than 1/2 -fe, in each channel, or 

(ii) d> is a pure phase shift, smaller than 60 — t degrees at all frequencies, in each channel. 

Proposition 1 showed that Theorem 3 generalizes the LQ gain and phase margin results of [13], 
The next proposition shows that the same Theorem generalizes the single-input results of [5] as well. 

Proposition 2: Suppose that h(y,) = i M u >) and le l be a memoryless nonlinearity such that 

*(n(-))(0 = faiMO.O. • • ••<M« m (0.0) r (19) 
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u,<Pi(u,) > uj — (1 — t) 


uMUj) 
&(«.) ’ 


( 20 ) 


Moreover, assume that u,(#i/du t )(u t ) > 0, Vu^Q. Then assumption 6 is satisfied. 


Proposition 2 may provide us with gain reduction margin results. As an application, let /ij(uj) *= 
uj n , for some positive integer n. Inequality (20) becomes 

( 21 ) 

which shows that the stronger we penalize large inputs (n large), the worse become the gain reduction 
margins, as should be expected. 


iy^ Q NCLvsms , 

This paper demonstrates that under suitable assumptions nonlinear optimal multi-input deter- 
ministic or stochastic dynamic systems have certain guaranteed robustness properties, which may be 
expressed as guaranteed gain and phase margins. These properties generalize the known robustness 
results of optimal regulators for linear systems with respect to quadratic performance criteria. In 
particular it is shown that if the nonlinear dynamic system is linear in the control variables and there 
is a quadratic penalty on the control variables in the associated cost functional, then the resulting 
nonlinear feedback design has a guaranteed infinite positive gain margin, a -6db gain reduction 
margin, and a ±60° phase margin property. 

Such guaranteed robustness properties are obtained from the Hamilton- Jacobi-Bellman equation 
associated with the nonlinear optimal control problems. 


t '.v ■ 
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This appendix contains all proofs for section m. 

Lemma 1: Case A: Given some z(Q)€H n and a time function u(t), let z(t) be the corresponding solution 
of (1), assuming that it exists. Assume hat /*[/(!( r)) + /»(tt(r))] dr < oo. Then, lim ( _, 00 :r;(t) *® 0. 
Case B: Given some z(0) and a stochastic process a(t), adapted to 7 () let i[t) be the corresponding 
solution of (5), assuming that it exists. Assume that / 0 a °[f(z(i r )) -f h(u(r))] dr < oo, almost surely, 
and that sup, V(z(t)) < oo, almost surely. Then limi—*!^) = 0, almost surely. 

Proof: Case A: Let V'(t) = /j W (l(l(f)) + h(si(r)))dr. Then, V(<)> V(s(t)), since V is the optimal cost- 
to-go function. Clearly, limj—^ V(f) — 0, which implies that 1 .n ( -.oc V(i(t)) = 0. By assumption 
la, it follows that lim e _ l(0 — 0. 

Case B: a) We first consider tl e case where 


E 



(i(l(r))+/i(l4(r)))d 


r 


< oo 


also holds. With V(t) defined as for Case A, it is easy to see that linit_ ae £[f'(f) |9<] = 0, a s . 
Moreover, the definition of V implies that £[^(0 which shows that V(z(t)) converges 

to zero and, using assumption lb, z(t) must converge to the origin, 
b) We now consider the general case. Given the initial state i(0) and some A/>0, N> 0 let 


Tv,. v = inf jf>0: j Q (l(l{r))+h{u(T)))dT>N or V(z(t))>Af|. 


If the above set it empty, let 7\f/v = oo. We now define a new control law & by 

u( 0 = tt(0i * < Tmn 
t>T MN , 

where &( ■) is an optimal control law and let £( } be the trajectory that results when & is used. Then, 



(f(£(r)) f M«(r)))d 



(l(l(r))+^r)))d 


+ F(V(z(Tmn)))<A/ + N. 


Therefore, by part (a), lim t _oo£(0 ** almost surely. Let Qmn — {uGO: Tmn — oo}. For all 
wEOm/v we have |{t) = z(t), it. Hence, lim t _ooZ(0 = 0, for almost all uj^Q MNi iM, N. On the 
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other hand, the assumptions of the Lemma imply that almost all u i&Sl also belong to Uv/\» for some 
M, iV, and the desired result follow$.| 

Proof of Theorem 1: We follow the approach of [5]. Fix some iGR n . Then, by assumption 3, 
dV /di{£) ■ l(i, y) is either increasing or decreasing, as a function of u,. Assume it is increasing. From 
either equation (4), (9) or (11), corresponding to cases A, B, C, respectively, we obtain 

^(i) • ill, k(i )) 4- Kk(z))<^li) ■ Liz, it) + %). (- 41 ) 

for all a that differ from fc(i) in the i-th component only. It follows that fci(i)<0 because otherwise 

h, (kill)) > hi{0) 


and 

where k,*{i) = 0, kffc) = kjfc), jj^i, which would contradict (Al). Similarly, we conclude that 
fc,(i)>0, whenever <9V/<9 j(j) • £fc,u) is decreasing as a function of u,-. 

Assumption 5 implies that <t>i{k,{i),t)>k t [i) whenever ^(;e)> 0 and <Pi{ki{i), t)<ki{i), otherwise. 
Together with the preceding discussion we conclude that 


^(l) • f(z, #&(*), 0) < /(*■ ttz))- 


(A2) 


From now on, let f[t) denote the trajectory of the perturbed closed-loop system and let u(t) = 

mm- 

Case A: (Deterministic Problems) From inequality (A2) and equation (4) v»e obtain 


^(i(0) = am. Mt). o) < • am. iwoi) = — ; (i(0) - M4wo))<o. (.«) 


Integrating (^ 3), we obtain 


/ ftl(r)) -I- hmT)))} dr < V(i(0)) - V(i(t)) < V(*(0)) 
Jo 
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and therefore, /' [f(j(r)) + /i (/c(s(r)))] dr < as. In view of assumption 2, the last inequality also 
gives /„* h(u(r))dr < oo. The desired result then follows from Lemma 1. 

Case B. (Stochastic Total Cost Problems) From inequality (A2) and equation (9) we obtain 

(ti‘ ( 'V)(i((i)<(i.i« , ' , v')(i(i)) = _((*(«)) - mm)- M4> 


Applying the Ito formula [15] to (A4), it follows that V(z(f)) is a positive supermartingale, converges 
almost surely [11] to a rand<\ i variable and, in particular, sup, V(x(f)) < oc, almost surely. The 
Ito formula also yields 


\i: 


(<<!(>■)) +M4({(r))))<Jr 


<V(!(0)) -E[V( I (())]<V'( I (0)). 


(/15) 


Taking the limit, as t—* oo, we obtain J|, (i(x(r)) ■+ h(k(x(r)))) dr < oo, almost surely. Then invoke 
assumption 2 and use Lemma 1 (as in the proof for Case A) to complete the proof. 

Case C: (Stochastic Average Cost Problems) Similarly with (A5) we obtain 

- 9t<V{x{0)) - E\V{x{t))]<V[x<Q)). (A6) 

Dividing by t and using Fatou’s Lemma [15] we obtain 



1 , 


1 /■' 

lim inf - / f(z(r))dr 

<— oc t Jo 

< lim inf E 

7 / *(*(*■)) dr 
t Jo 

t—oc. 


<9- 


Therefore, lim inft_ 00 (l/f) /J /(j(t)) dr < oo, almost surely, which, in view of Assumption lc, implies 
the last part of the theorem. | 

Proof of Theorem 2: The proof of sufficiency is trivial, so we concentrate on the proof of necessity. 
Since the theorem has to be proved for each z separately, we assume that a particular z has been 
fixed and we drop the dependence on z from our notation. 

Assume, without loss of generality, that there is no i such that Q //<9u,(u) is identically zero. 
For any i, let Hi = {dl/du,(u): u£R m ] . If there exist u 1 , y 2 such that df/du t (y l ) and <9//<9u,(jx 2 ) are 


14 



ORIGINAL PAGfc IS 
OF POOR QUALITY 


not collinear, then there exists a vector ft such that dfjdu,{ it 1 ) • ft > 0 and <9£/0u,(ft 2 ) • ft < 0, thus 
contradicting assumption 3(ii). Therefore, Hi is contained in some one-dimensional subspace of R m , 
which may be repersented by some nonzero vector (>,£//,. 

We now partition the set { 1, . . m} of components of u into a set of classes A\, . . Aq, as follows: 
two components i, j will belong to the same class if and only if fij is collinear to 

Since <9//<9u, is collinear to b h so must b t&f/duiduj. By interchanging the order of differentiation, 
we conclude that either i and j belong to the same class, or d 2 f/duiduj(u) = 0, Vu. Based on 
this observation, the representation (17) follows immediately. The fact that the functions /* must 
themselves satisfy assumption 3(ii) is also straightforward.| 


Proof of Theorem 3: From any one of the formulae (4), (9) or (11), corresponding to cases A, B, 
C, respectively, we obtair, by differentiating with respect to u, 


!)«*» + f£fc) • m - o. 


(/t7) 


Let x(t), u(£) be the trajectories of the state and the control resulting f r om the perturbed closed- 
loop system. Let, for convenience, 


=^(H*(0))(*(«*(-))X«) -HiWII + K4W9)) = 
- fj(iW)ni«>iMi(0) - )))(<)] + Km))). 


(A8) 


where the last equality following from (A7). Note that assumption 6(ii) states that 


f c(r)dr>e [ h(k(x(T)))dT, Vf>0. 

Jo Jo 


m 


Case A: (Deterministic Problems) Using equation (4) and (A8) 


~m ) = = 

= ■ cm) + %mvm)m)) + mm) - = 

OX ox 

= —W)) — c(f). 


(A10) 
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We now integrate (A10), use (A9) and take the limit as t — *oo to obtain 

(*U( r )) + <M*te(r)))) < oo 


r< 


and the desired result follows from Lemma 1, as in the proof of Theorem 1. 

Case B: (Stochastic Total Cost Problems) Similarly with (A10), we obtain from (9) and (A8), 

We now integrate (All) and use the Ito rule to obtain, for 0<t<T, 

rT 


(Aii) 


E[V(i(T))|5,]-V'( I (()) 

Given that c(t) is adapted to {^ ( }, it follows that 


- -it 


c(T)dr |®F, 


Vfe(7))+ f c[T)dT\% <VWO)+ / c(r)di 
Jo Jo 


which shows that V(^(t)) / Q f c(r)dr is a (positive) supermartingale and therefore converges Hence 
V(j(f)) has bounded sample paths. From (A9), (All) and the Ito rule we obtain 


[(W) 

Jo 


+ th[k(z(T))))dT 


<E 


fmr)) 

Jo 


4 - dj))dr 


< ^(1(0)), V<>0. 


(412) 


In view of assumption 2, / 0 °°(f(x(r)) J-h(j 4 (r)))dr < oo, and the desired result follows from Lemma 

1. 

Case C: (Stochastic Average Cost Problems) Similarly with (A12) we obtain 


E / l(z{r))dT 
Jo 


<V(z<,) + gt, Vt> 0. 


This is the same inequality as (A6) in the proof of Theorem 1 and the rest of the proof is the same 
as in Theorem l.| 


Proof of Proposition 1: The proof of the first statement is immediate from ParsevaFs theorem. See 
also [13, p. 177]. The next statements are Corollaries 4 and 5 of [13].| 


Proof of Proposition 2: It is trivial to check that the integrand in the left hand side of (18) will be 
nonnegative for all r; hence (18) is satisfied.) 
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FIG. 1: The Perturbed Closed-Loop System. 



FIG. 2: 


A Memoryless Gain Increase. 
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